The effectiveness of pay-for-performance contracts with non-governmental organizations in Afghanistan - results of a controlled interrupted time series analysis.

BACKGROUND: In many contexts, including fragile settings like Afghanistan, the coverage of basic health services is low. To address these challenges there has been considerable interest in working with NGOs and examining the effect of financial incentives on service providers. The Government of Afghanistan has used contracting with NGOs for more than 15 years and in 2019 introduced pay-for-performance (P4P) into the contracts. This study examines the impact of P4P on health service delivery in Afghanistan. METHODS: We conducted an interrupted time series (ITS) analysis with a non-randomized comparison group that employed segmented regression models and used independently verified health management information system (HMIS) data from 2015 to 2021. We compared 31 provinces with P4P contracts to 3 provinces where the Ministry of Public Health (MOPH) continued to deliver services without P4P. We used data from annual health facility surveys to assess the quality of care. FINDINGS: Independent verification of the HMIS data found that consistency and accuracy was greater than 90% in the contracted provinces. The introduction of P4P increased the 10 P4P-compensated service delivery outcomes by a median of 22.1 percentage points (range 10.2 to 43.8) for the two-arm analysis and 19.9 percentage points (range: - 8.3 to 56.1) for the one-arm analysis. There was a small decrease in quality of care initially, but it was short-lived. We found few other unintended consequences. INTERPRETATION: P4P contracts with NGOs led to a substantial improvement in service delivery at lower cost despite a very difficult security situation. The promising results from this large-scale experience warrant more extensive application of P4P contracts in other fragile settings or wherever coverage remains low.

Surveillance as a Core Intervention to Strengthen Malaria Control Programs in Moderate to High Transmission Settings.

New tools are needed for malaria control, and recent improvements in malaria surveillance have opened the possibility of transforming surveillance into a core intervention. Implementing this strategy can be challenging in moderate to high transmission settings. However, there is a wealth of practical experience among national malaria control programs and partners working to improve and use malaria surveillance data to guide programming. Granular and timely data are critical to understanding geographic heterogeneity, appropriately defining and targeting interventions packages, and enabling timely decision-making at the operational level. Resources to be targeted based on surveillance data include vector control, case management commodities, outbreak responses, quality improvement interventions, and human resources, including community health workers, as they contribute to a more refined granularity of the surveillance system. Effectively transforming malaria surveillance into a core intervention will require strong global and national leadership, empowerment of subnational and local leaders, collaboration among development partners, and global coordination. Ensuring that national health systems include community health work can contribute to a successful transformation. It will require a strong supply chain to ensure that all suspected cases can be diagnosed and data reporting tools including appropriate electronic devices to provide timely data. Regular data quality audits, decentralized implementation, supportive supervision, data-informed decision-making processes, and harnessing technology for data analysis and visualization are needed to improve the capacity for data-driven decision-making at all levels. Finally, resources must be available to respond programmatically to these decisions.

The effectiveness of supervision strategies to improve health care provider practices in low- and middle-income countries: secondary analysis of a systematic review.

BACKGROUND: Although supervision is a ubiquitous approach to support health programs and improve health care provider (HCP) performance in low- and middle-income countries (LMICs), quantitative evidence of its effects is unclear. The objectives of this study are to describe the effect of supervision strategies on HCP practices in LMICs and to identify attributes associated with greater effectiveness of routine supervision. METHODS: We performed a secondary analysis of data on HCP practice outcomes (e.g., percentage of patients correctly treated) from a systematic review on improving HCP performance. The review included controlled trials and interrupted time series studies. We described distributions of effect sizes (defined as percentage-point [%-point] changes) for each supervision strategy. To identify attributes associated with supervision effectiveness, we performed random-effects linear regression modeling and examined studies that directly compared different approaches of routine supervision. RESULTS: We analyzed data from 81 studies from 36 countries. For professional HCPs, such as nurses and physicians, primarily working at health facilities, routine supervision (median improvement when compared to controls: 10.7%-points; IQR: 9.9, 27.9) had similar effects on HCP practices as audit with feedback (median improvement: 10.1%-points; IQR: 6.2, 23.7). Two attributes were associated with greater mean effectiveness of routine supervision (p < 0.10): supervisors received supervision (by 8.8-11.5%-points), and supervisors participated in problem-solving with HCPs (by 14.2-20.8%-points). Training for supervisors and use of a checklist during supervision visits were not associated with effectiveness. The effects of supervision frequency (i.e., number of visits per year) and dose (i.e., the number of supervision visits during a study) were unclear. For lay HCPs, the effect of routine supervision was difficult to characterize because few studies existed, and effectiveness in those studies varied considerably. Evidence quality for all findings was low primarily because many studies had a high risk of bias. CONCLUSIONS: Although evidence is limited, to promote more effective supervision, our study supports supervising supervisors and having supervisors engage in problem-solving with HCPs. Supervision's integral role in health systems in LMICs justifies a more deliberate research agenda to identify how to deliver supervision to optimize its effect on HCP practices.

How does the effectiveness of strategies to improve healthcare provider practices in low-income and middle-income countries change after implementation? Secondary analysis of a systematic review.

BACKGROUND: A recent systematic review evaluated the effectiveness of strategies to improve healthcare provider (HCP) performance in low-income and middle-income countries. The review identified strategies with varying effects, including in-service training, supervision and group problem-solving. However, whether their effectiveness changed over time remained unclear. In particular, understanding whether effects decay over time is crucial to improve sustainability. METHODS: We conducted a secondary analysis of data from the aforementioned review to explore associations between time and effectiveness. We calculated effect sizes (defined as percentage-point (%-point) changes) for HCP practice outcomes (eg, percentage of patients correctly treated) at each follow-up time point after the strategy was implemented. We estimated the association between time and effectiveness using random-intercept linear regression models with time-specific effect sizes clustered within studies and adjusted for baseline performance. RESULTS: The primary analysis included 37 studies, and a sensitivity analysis included 77 additional studies. For training, every additional month of follow-up was associated with a 0.19 %-point decrease in effectiveness (95% CI: -0.36 to -0.03). For training combined with supervision, every additional month was associated with a 0.40 %-point decrease in effectiveness (95% CI: -0.68 to -0.12). Time trend results for supervision were inconclusive. For group problem-solving alone, time was positively associated with effectiveness, with a 0.50 %-point increase in effect per month (95% CI: 0.37 to 0.64). Group problem-solving combined with training was associated with large improvements, and its effect was not associated with time. CONCLUSIONS: Time trends in the effectiveness of different strategies to improve HCP practices vary among strategies. Programmes relying solely on in-service training might need periodical refresher training or, better still, consider combining training with group problem-solving. Although more high-quality research is needed, these results, which are important for decision-makers as they choose which strategies to use, underscore the utility of studies with multiple post-implementation measurements so sustainability of the impact on HCP practices can be assessed.

Opening the 'black box' of collaborative improvement: a qualitative evaluation of a pilot intervention to improve quality of malaria surveillance data in public health centres in Uganda.

BACKGROUND: Demand for high-quality surveillance data for malaria, and other diseases, is greater than ever before. In Uganda, the primary source of malaria surveillance data is the Health Management Information System (HMIS). However, HMIS data may be incomplete, inaccurate or delayed. Collaborative improvement (CI) is a quality improvement intervention developed in high-income countries, which has been advocated for low-resource settings. In Kayunga, Uganda, a pilot study of CI was conducted in five public health centres, documenting a positive effect on the quality of HMIS and malaria surveillance data. A qualitative evaluation was conducted concurrently to investigate the mechanisms of effect and unintended consequences of the intervention, aiming to inform future implementation of CI. METHODS: The study intervention targeted health workers, including brief in-service training, plus CI with 'plan-do-study-act' (PDSA) cycles emphasizing self-reflection and group action, periodic learning sessions, and coaching from a CI mentor. Health workers collected data on standard HMIS out-patient registers. The qualitative evaluation (July 2015 to September 2016) included ethnographic observations at each health centre (over 12-14 weeks), in-depth interviews with health workers and stakeholders (n = 20), and focus group discussions with health workers (n = 6). RESULTS: The results suggest that the intervention did facilitate improvement in data quality, but through unexpected mechanisms. The CI intervention was implemented as planned, but the PDSA cycles were driven largely by the CI mentor, not the health workers. In this context, characterized by a rigid hierarchy within the health system of limited culture of self-reflection and inadequate training and supervision, CI became an effective form of high-quality training with frequent supervisory visits. Health workers appeared motivated to improve data collection habits by their loyalty to the CI mentor and the potential for economic benefits, rather than a desire for self-improvement. CONCLUSIONS: CI is a promising method of quality improvement and could have a positive impact on malaria surveillance data. However, successful scale-up of CI in similar settings may require deployment of highly skilled mentors. Further research, focusing on the effectiveness of 'real world' mentors using robust study designs, will be required to determine whether CI can be translated effectively and sustainably to low-resource settings.

Effectiveness of in-service training plus the collaborative improvement strategy on the quality of routine malaria surveillance data: results of a pilot study in Kayunga District, Uganda.

BACKGROUND: Surveillance data are essential for malaria control, but quality is often poor. The aim of the study was to evaluate the effectiveness of the novel combination of training plus an innovative quality improvement method-collaborative improvement (CI)-on the quality of malaria surveillance data in Uganda. METHODS: The intervention (training plus CI, or TCI), including brief in-service training and CI, was delivered in 5 health facilities (HFs) in Kayunga District from November 2015 to August 2016. HF teams monitored data quality, conducted plan-do-study-act cycles to test changes, attended periodic learning sessions, and received CI coaching. An independent evaluation was conducted to assess data completeness, accuracy, and timeliness. Using an interrupted time series design without a separate control group, data were abstracted from 156,707 outpatient department (OPD) records, laboratory registers, and aggregated monthly reports (MR) for 4 time periods: baseline-12 months, TCI scale-up-5 months; CI implementation-9 months; post-intervention-4 months. Monthly OPD register completeness was measured as the proportion of patient records with a malaria diagnosis with: (1) all data fields completed, and (2) all clinically-relevant fields completed. Accuracy was the relative difference between: (1) number of monthly malaria patients reported in OPD register versus MR, and (2) proportion of positive malaria tests reported in the laboratory register versus MR. Data were analysed with segmented linear regression modelling. RESULTS: Data completeness increased substantially following TCI. Compared to baseline, all-field completeness increased by 60.1%-points (95% confidence interval [CI]: 46.9-73.2%) at mid-point, and clinically-relevant completeness increased by 61.6%-points (95% CI: 56.6-66.7%). A relative - 57.4%-point (95% confidence interval: - 105.5, - 9.3%) change, indicating an improvement in accuracy of malaria test positivity reporting, but no effect on data accuracy for monthly malaria patients, were observed. Cost per additional malaria patient, for whom complete clinically-relevant data were recorded in the OPD register, was $3.53 (95% confidence interval: $3.03, $4.15). CONCLUSIONS: TCI improved malaria surveillance completeness considerably, with limited impact on accuracy. Although these results are promising, the intervention's effectiveness should be evaluated in more HFs, with longer follow-up, ideally in a randomized trial, before recommending CI for wide-scale use.

The effectiveness of training strategies to improve healthcare provider practices in low-income and middle-income countries.

INTRODUCTION: In low/middle-income countries (LMICs), training is often used to improve healthcare provider (HCP) performance. However, important questions remain about how well training works and the best ways to design training strategies. The objective of this study is to characterise the effectiveness of training strategies to improve HCP practices in LMICs and identify attributes associated with training effectiveness. METHODS: We performed a secondary analysis of data from a systematic review on improving HCP performance. The review included controlled trials and interrupted time series, and outcomes measuring HCP practices (eg, percentage of patients correctly treated). Distributions of effect sizes (defined as percentage-point (%-point) changes) were described for each training strategy. To identify effective training attributes, we examined studies that directly compared training approaches and performed random-effects linear regression modelling. RESULTS: We analysed data from 199 studies from 51 countries. For outcomes expressed as percentages, educational outreach visits (median effect size when compared with controls: 9.9 %-points; IQR: 4.3-20.6) tended to be somewhat more effective than in-service training (median: 7.3 %-points; IQR: 3.6-17.4), which seemed more effective than peer-to-peer training (4.0 %-points) and self-study (by 2.0-9.3 %-points). Mean effectiveness was greater (by 6.0-10.4 %-points) for training that incorporated clinical practice and training at HCPs' work site. Attributes with little or no effect were: training with computers, interactive methods or over multiple sessions; training duration; number of educational methods; distance training; trainers with pedagogical training and topic complexity. For lay HCPs, in-service training had no measurable effect. Evidence quality for all findings was low. CONCLUSIONS: Although additional research is needed, by characterising the effectiveness of training strategies and identifying attributes of effective training, decision-makers in LMICs can improve how these strategies are selected and implemented.

The effectiveness of the quality improvement collaborative strategy in low- and middle-income countries: A systematic review and meta-analysis.

BACKGROUND: Quality improvement collaboratives (QICs) have been used to improve health care for decades. Evidence on QIC effectiveness has been reported, but systematic reviews to date have little information from low- and middle-income countries (LMICs). OBJECTIVE: To assess the effectiveness of QICs in LMICs. METHODS: We conducted a systematic review following Cochrane methods, the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach for quality of evidence grading, and the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting. We searched published and unpublished studies between 1969 and March 2019 from LMICs. We included papers that compared usual practice with QICs alone or combined with other interventions. Pairs of reviewers independently selected and assessed the risk of bias and extracted data of included studies. To estimate strategy effectiveness from a single study comparison, we used the median effect size (MES) in the comparison for outcomes in the same outcome group. The primary analysis evaluated each strategy group with a weighted median and interquartile range (IQR) of MES values. In secondary analyses, standard random-effects meta-analysis was used to estimate the weighted mean MES and 95% confidence interval (CI) of the mean MES of each strategy group. This review is registered with PROSPERO (International Prospective Register of Systematic Reviews): CRD42017078108. RESULTS: Twenty-nine studies were included; most (21/29, 72.4%) were interrupted time series studies. Evidence quality was generally low to very low. Among studies involving health facility-based health care providers (HCPs), for "QIC only", effectiveness varied widely across outcome groups and tended to have little effect for patient health outcomes (median MES less than 2 percentage points for percentage and continuous outcomes). For "QIC plus training", effectiveness might be very high for patient health outcomes (for continuous outcomes, median MES 111.6 percentage points, range: 96.0 to 127.1) and HCP practice outcomes (median MES 52.4 to 63.4 percentage points for continuous and percentage outcomes, respectively). The only study of lay HCPs, which used "QIC plus training", showed no effect on patient care-seeking behaviors (MES -0.9 percentage points), moderate effects on non-care-seeking patient behaviors (MES 18.7 percentage points), and very large effects on HCP practice outcomes (MES 50.4 percentage points). CONCLUSIONS: The effectiveness of QICs varied considerably in LMICs. QICs combined with other invention components, such as training, tended to be more effective than QICs alone. The low evidence quality and large effect sizes for QIC plus training justify additional high-quality studies assessing this approach in LMICs.

A systematic review of the effectiveness of strategies to improve health care provider performance in low- and middle-income countries: Methods and descriptive results.

BACKGROUND: Health care provider (HCP) performance in low- and middle-income countries (LMICs) is often inadequate. The Health Care Provider Performance Review (HCPPR) is a comprehensive systematic review of the effectiveness and cost of strategies to improve HCP performance in LMICs. We present the HCPPR's methods, describe methodological and contextual attributes of included studies, and examine time trends of study attributes. METHODS: The HCPPR includes studies from LMICs that quantitatively evaluated any strategy to improve HCP performance for any health condition, with no language restrictions. Eligible study designs were controlled trials and interrupted time series. In 2006, we searched 15 databases for published studies; in 2008 and 2010, we completed searches of 30 document inventories for unpublished studies. Data from eligible reports were double-abstracted and entered into a database, which is publicly available. The primary outcome measure was the strategy's effect size. We assessed time trends with logistic, Poisson, and negative binomial regression modeling. We were unable to register with PROSPERO (International Prospective Register of Systematic Reviews) because the protocol was developed prior to the PROSPERO launch. RESULTS: We screened 105,299 citations and included 824 reports from 499 studies of 161 intervention strategies. Most strategies had multiple components and were tested by only one study each. Studies were from 79 countries and had diverse methodologies, geographic settings, HCP types, work environments, and health conditions. Training, supervision, and patient and community supports were the most commonly evaluated strategy components. Only 33.6% of studies had a low or moderate risk of bias. From 1958-2003, the number of studies per year and study quality increased significantly over time, as did the proportion of studies from low-income countries. Only 36.3% of studies reported information on strategy cost or cost-effectiveness. CONCLUSIONS: Studies have reported on the efficacy of many strategies to improve HCP performance in LMICs. However, most studies have important methodological limitations. The HCPPR is a publicly accessible resource for decision-makers, researchers, and others interested in improving HCP performance.

The Effect of Text Message Reminders to Health Workers on Quality of Care for Malaria, Pneumonia, and Diarrhea in Malawi: A Cluster-Randomized, Controlled Trial.

The use of mobile technologies in medicine, or mHealth, holds promise to improve health worker (HW) performance, but evidence is mixed. We conducted a cluster-randomized controlled trial to evaluate the effect of text message reminders to HWs in outpatient health facilities (HFs) on quality of care for malaria, pneumonia, and diarrhea in Malawi. After a baseline HF survey (2,360 patients) in January 2015, 105 HFs were randomized to three arms: 1) text messages to HWs on malaria case management; 2) text messages to HWs on malaria, pneumonia, and diarrhea case management (latter two for children < 5 years); and 3) control arm (no messages). Messages were sent beginning April 2015 twice daily for 6 months, followed by an endline HF survey (2,536 patients) in November 2015. An intention-to-treat analysis with difference-in-differences binomial regression modeling was performed. The proportion of patients with uncomplicated malaria managed correctly increased from 42.8% to 59.6% in the control arm, from 43.7% to 55.8% in arm 1 (effect size -4.7%-points, 95% confidence interval (CI): -18.2, 8.9, P = 0.50) and from 30.2% to 50.9% in arm 2 (effect size 3.9%-points, 95% CI: -14.1, 22.0, P = 0.67). Prescription of first-line antibiotics to children < 5 years with clinically defined pneumonia increased in all arms, but decreased in arm 2 (effect size -4.1%-points, 95% CI: -42.0, 33.8, P = 0.83). Prescription of oral rehydration solution to children with diarrhea declined slightly in all arms. We found no significant improvements in malaria, pneumonia, or diarrhea treatment after HW reminders, illustrating the importance of rigorously testing new interventions before adoption.

The effect of mobile phone text message reminders on health workers' adherence to case management guidelines for malaria and other diseases in Malawi: lessons from qualitative data from a cluster-randomized trial.

BACKGROUND: Mobile health (mHealth), which uses technology such as mobile phones to improve patient health and health care delivery, is increasingly being tested as an intervention to promote health worker (HW) performance. This study assessed the effect of short messaging services (SMS) reminders in a study setting. Following a trial of text-message reminders to HWs to improve case management of malaria and other childhood diseases in southern Malawi that showed little effect, qualitative data was collected to explore the reasons why the intervention was ineffective and describe lessons learned. METHODS: Qualitative data collection was undertaken to lend insight into quantitative results from a trial in which 105 health facilities were randomized to three arms: (1) twice-daily text-message reminders to HWs, including clinicians and drug dispensers, on case management of malaria; (2) twice-daily text-message reminders to HWs on case management of malaria, pneumonia and diarrhoea; and, (3) a control arm. In-depth interviews were conducted with 50 HWs in the intervention arms across seven districts. HWs were asked about acceptability and feasibility of the text-messaging intervention and its perceived impact on recommended case management. The interviews were recorded, transcribed and translated into English for a thematic and framework analysis. Nvivo 11 software was used for data management and analysis. RESULTS: A total of 50 HWs were interviewed at 22 facilities. HWs expressed high acceptance of text-message reminders and appreciated messages as job aids and practical reference material for their day-to-day work. However, HWs said that health systems barriers, including very high outpatient workload, commodity stock-outs, and lack of supportive supervision and financial incentives demotivated them, limited their ability to act on messages and therefore adherence to case management guidelines. Drug dispensers were more likely than clinicians to report usage of text-message reminders. Despite these challenges, nearly all HWs expressed a desire for a longer duration of the SMS intervention. CONCLUSIONS: Text-message reminders to HWs can provide a platform to improve understanding of treatment guidelines and case management decision-making skills, but might not improve actual adherence to guidelines. More interaction, for example through targeted supervision or two-way technology communication, might be an essential intervention component to help address structural barriers and facilitate improved clinical practice.

Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review.

BACKGROUND: Inadequate health-care provider performance is a major challenge to the delivery of high-quality health care in low-income and middle-income countries (LMICs). The Health Care Provider Performance Review (HCPPR) is a comprehensive systematic review of strategies to improve health-care provider performance in LMICs. METHODS: For this systematic review we searched 52 electronic databases for published studies and 58 document inventories for unpublished studies from the 1960s to 2016. Eligible study designs were controlled trials and interrupted time series. We only included strategy-versus-control group comparisons. We present results of improving health-care provider practice outcomes expressed as percentages (eg, percentage of patients treated correctly) or as continuous measures (eg, number of medicines prescribed per patient). Effect sizes were calculated as absolute percentage-point changes. The summary measure for each comparison was the median effect size (MES) for all primary outcomes. Strategy effectiveness was described with weighted medians of MES. This study is registered with PROSPERO, number CRD42016046154. FINDINGS: We screened 216 477 citations and selected 670 reports from 337 studies of 118 strategies. Most strategies had multiple intervention components. For professional health-care providers (generally, facility-based health workers), the effects were near zero for only implementing a technology-based strategy (median MES 1·0 percentage points, IQR -2·8 to 9·9) or only providing printed information for health-care providers (1·4 percentage points, -4·8 to 6·2). For percentage outcomes, training or supervision alone typically had moderate effects (10·3-15·9 percentage points), whereas combining training and supervision had somewhat larger effects than use of either strategy alone (18·0-18·8 percentage points). Group problem solving alone showed large improvements in percentage outcomes (28·0-37·5 percentage points), but, when the strategy definition was broadened to include group problem solving alone or other strategy components, moderate effects were more typical (12·1 percentage points). Several multifaceted strategies had large effects, but multifaceted strategies were not always more effective than simpler ones. For lay health-care providers (generally, community health workers), the effect of training alone was small (2·4 percentage points). Strategies with larger effect sizes included community support plus health-care provider training (8·2-125·0 percentage points). Contextual and methodological heterogeneity made comparisons difficult, and most strategies had low quality evidence. INTERPRETATION: The impact of strategies to improve health-care provider practices varied substantially, although some approaches were more consistently effective than others. The breadth of the HCPPR makes its results valuable to decision makers for informing the selection of strategies to improve health-care provider practices in LMICs. These results also emphasise the need for researchers to use better methods to study the effectiveness of interventions. FUNDING: Bill & Melinda Gates Foundation, CDC Foundation.

Infection prevention and control training and capacity building during the Ebola epidemic in Guinea.

BACKGROUND: During the 2014-2016 Ebola epidemic in West Africa, a key epidemiological feature was disease transmission within healthcare facilities, indicating a need for infection prevention and control (IPC) training and support. METHODS: IPC training was provided to frontline healthcare workers (HCW) in healthcare facilities that were not Ebola treatment units, as well as to IPC trainers and IPC supervisors placed in healthcare facilities. Trainings included both didactic and hands-on components, and were assessed using pre-tests, post-tests and practical evaluations. We calculated median percent increase in knowledge. RESULTS: From October-December 2014, 20 IPC courses trained 1,625 Guineans: 1,521 HCW, 55 IPC trainers, and 49 IPC supervisors. Median test scores increased 40% (interquartile range [IQR]: 19-86%) among HCW, 15% (IQR: 8-33%) among IPC trainers, and 21% (IQR: 15-30%) among IPC supervisors (all P<0.0001) to post-test scores of 83%, 93%, and 93%, respectively. CONCLUSIONS: IPC training resulted in clear improvements in knowledge and was feasible in a public health emergency setting. This method of IPC training addressed a high demand among HCW. Valuable lessons were learned to facilitate expansion of IPC training to other prefectures; this model may be considered when responding to other large outbreaks.

Practical recommendations for the evaluation of improvement initiatives.

A lack of clear guidance for funders, evaluators and improvers on what to include in evaluation proposals can lead to evaluation designs that do not answer the questions stakeholders want to know. These evaluation designs may not match the iterative nature of improvement and may be imposed onto an initiative in a way that is impractical from the perspective of improvers and the communities with whom they work. Consequently, the results of evaluations are often controversial, and attribution remains poorly understood. Improvement initiatives are iterative, adaptive and context-specific. Evaluation approaches and designs must align with these features, specifically in their ability to consider complexity, to evolve as the initiative adapts over time and to understand the interaction with local context. Improvement initiatives often identify broadly defined change concepts and provide tools for care teams to tailor these in more detail to local conditions. Correspondingly, recommendations for evaluation are best provided as broad guidance, to be tailored to the specifics of the initiative. In this paper, we provide practical guidance and recommendations that funders and evaluators can use when developing an evaluation plan for improvement initiatives that seeks to: identify the questions stakeholders want to address; develop the initial program theory of the initiative; identify high-priority areas to measure progress over time; describe the context the initiative will be applied within; and identify experimental or observational designs that will address attribution.

Quality of Case Management for Pneumonia and Diarrhea Among Children Seen at Health Facilities in Southern Malawi.

Pneumonia and diarrhea are leading causes of child deaths in Malawi. Guidelines to manage childhood illnesses in resource-poor settings exist, but studies have reported low health-care worker (HCW) adherence to guidelines. We conducted a health facility survey from January to March 2015 to assess HCW management of pneumonia and diarrhea in children < 5 years of age in southern Malawi, and to determine factors associated with case management quality. Descriptive statistics and multivariable logistic regression models examined patient, HCW, and health facility factors associated with recommended pneumonia and diarrhea management, using Malawi's national guidelines as the gold standard. Of 694 surveyed children 2-59 months of age at 95 health facilities, 132 (19.0%) met survey criteria for pneumonia; HCWs gave recommended antibiotic treatment to 90 (68.2%). Of 723 children < 5 years of age, 222 (30.7%) had uncomplicated diarrhea; HCWs provided recommended treatment to 94 (42.3%). In multivariable analyses, caregivers' spontaneous report of children's symptoms was associated with recommended treatment of both pneumonia (odds ratio [OR]: 2.8, 95% confidence interval [CI]: 1.2-6.8, P = 0.023) and diarrhea (OR: 24.2, 95% CI: 6.0-97.0, P < 0001). Malaria diagnosis was negatively associated with recommended treatment (OR for pneumonia: 0.5, 95% CI: 0.2-1.0, P = 0.046; OR for diarrhea: 0.3, 95% CI: 0.1-0.6, P = 0.003). To improve quality of care, children should be assessed systematically, even when malaria is suspected. Renewed efforts to invigorate such a systematic approach, including HCW training, regular follow-up supervision, and monitoring HCW performance, are needed in Malawi.

Health worker adherence to malaria treatment guidelines at outpatient health facilities in southern Malawi following implementation of universal access to diagnostic testing.

BACKGROUND: Appropriate diagnosis and treatment are essential for reducing malaria mortality. A cross-sectional outpatient health facility (HF) survey was conducted in southern Malawi from January to March 2015 to determine appropriate malaria testing and treatment practices four years after implementation of a policy requiring diagnostic confirmation before treatment. METHODS: Enrolled patients were interviewed, examined and had their health booklet reviewed. Health workers (HWs) were asked about training, supervision and access to the 2013 national malaria treatment guidelines. HFs were assessed for malaria diagnostic and treatment capacity. Weighted descriptive analyses and logistic regression of patient, HW and HF characteristics related to testing and treatment were performed. RESULTS: An evaluation of 105 HFs, and interviews of 150 HWs and 2342 patients was completed. Of 1427 suspect uncomplicated malaria patients seen at HFs with testing available, 1072 (75.7%) were tested, and 547 (53.2%) tested positive. Testing was more likely if patients spontaneously reported fever (odds ratio (OR) 2.6; 95% confidence interval (CI) 1.7-4.0), headache (OR 1.5; 95% CI 1.1-2.1) or vomiting (OR 2.0; 95% CI 1.0-4.0) to HWs and less likely if they reported skin problems (OR 0.4; 95% CI 0.2-0.6). Altogether, 511 (92.7%) confirmed cases and 98 (60.3%) of 178 presumed uncomplicated malaria patients (at HFs without testing) were appropriately treated, while 500 (96.6%) of 525 patients with negative tests did not receive anti-malarials. Only eight (5.7%) suspect severe malaria patients received appropriate pre-referral treatment. Appropriate treatment was more likely for presumed uncomplicated malaria patients (at HFs without testing) with elevated temperature (OR 1.5/1 °C increase; 95% CI 1.1-1.9), who reported fever to HWs (OR 5.7; 95% CI 1.9-17.6), were seen by HWs with additional supervision visits in the previous 6 months (OR 1.2/additional visit; 95% CI 1.0-1.4), or were seen by older HWs (OR 1.1/year of age; 95% CI 1.0-1.1). CONCLUSIONS: Correct testing and treatment practices were reasonably good for uncomplicated malaria when testing was available. Pre-referral treatment for suspect severe malaria was unacceptably rare. Encouraging HWs to elicit and appropriately respond to patient symptoms may improve practices.

Predictors of health worker performance after Integrated Management of Childhood Illness training in Benin: a cohort study.

BACKGROUND: Correct treatment of potentially life-threatening illnesses (PLTIs) in children under 5 years, such as malaria, pneumonia, and diarrhea, can substantially reduce mortality. The Integrated Management of Childhood Illness (IMCI) strategy has been shown to improve treatment of child illnesses, but multiple studies have shown that gaps in health worker performance remain after training. To better understand factors related to health worker performance, we analyzed 9,330 patient consultations in Benin from 2001-2002, after training one of the first cohorts of 32 health workers in IMCI. METHODS: With data abstracted from patient registers specially designed for IMCI-trained health workers, we examined associations between health facility-, health worker-, and patient-level factors and 10 case-management outcomes for PLTIs. RESULTS: Altogether, 63.6% of children received treatment for all their PLTIs in accordance with IMCI guidelines, and 77.8% received life-saving treatment (i.e., clinically effective treatment, even if not exactly in accordance with IMCI guidelines). Performance of individual health workers varied greatly, from 15-88% of patients treated correctly, on average. Multivariate regression analyses identified several factors that might have influenced case-management quality, many outside a manager's direct control. Younger health workers significantly outperformed older ones, and infants received better care than older children. Children with danger signs, those with more complex illnesses, and those with anemia received worse care. Health worker supervision was associated with improved performance for some outcomes. CONCLUSIONS: A variety of factors, some outside the direct control of program managers, can influence health worker practices. An understanding of these influences can help inform the development of strategies to improve performance.